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Pi. 

.'clopment  of  ti me-shered  computer  syste=T-:s  1/    has 
1    to  major  technical  and  philosophic  changes  wit:-, in  the 
cc  puter  field  during  this  decade. 1/   Computer  manuif acturers 
ho  3  available  both  computer  hardware  ^\-\(\    software  systems 
specially  designed  for  time-sharing  usage.   The  Adv,anced 
Research  Projects  Agency  (ARPA)  of  the  Pentagon  h5:i  given 
Increasing  amounts  of  support  to  M.KT.  and  other  ^^.-ademic 
(and  non-profit)  communities  for  research  into  t  i  mu- -shar i ng. 
The  Joint  Computer  Confei'ences  sponsored  by  AF I  PS  ^/    has 
devoted  an  Increasing  number  of  sessions  at  their  r.cetlngs 
to  tims-shar i ng. 

•  The  primary  concern  voiced  in  these  meetings  r- nd  the 
published  literature  appears  to  center  around  proL-e;ns  of 
the  design  and  development  of  the  ti  me- slir^red  comp-ter 
system  --  little  or  no  interest  seems  to  hr.ve  been  generated 
for  ^\^    evaluation  of  these  systems  as  to  their  usefulness 
for  present  or  future  users. 

The  need  for  such  a  formal  investigation  into  the 
usefulness  of  time -sharing  is  hi  ghl  I  gli  tr.rl  by:  (1)  :;ie  lack 
of  Information  concerning  the  value  of  the  rJ.'iM*iLflI.iXLsJtjjr-.5. 
of  time-sharing  computer  systems  v>'hich  mc'!;es  these  systems 
differ  from  the  more  traditional  ba  tch-pi'cccr.s  I  ng  computer 
systems;  and  (2)  uncertainty  as  to  the  urcfulncrs  o? 
time-sharing  systems  or  the  premium  (If  any)  tliat  mtist  be 
paid  for  this  service. 1l/  This  proposed  research  will 
evaluate  time-shared  computer  system  usage  by  means  of  a 
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;   -rn         i nv ostl sat  Ion    Into    the    relationship    between    user 

rf  :/    behavloi*/    and   Cittltudes    and    the    characteristics 

^       b^T'h    tiie    time-shared   computer    systems   and    the    comparable 
c    nventlonal    computer   systems. 
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R,    M.  Fano  has  stated/  "The  notion  of  machine-aided 

cognition  implies  an  intimate  collaboration  betv^/een  a  human 

user  and  a  computer  in  a  real-=time  dialogue  on  the  solution 

Q?    a  problem,  in  which  the  two  parties  contribute  their  best 

capabilities.  .  .  ".  (16)  Others  with  considerable 

experience  in  this  area  have  also  added  their  support  to  the 

development  of  interactive  time-shared  computer  systems  to 

aid    In    this   man-machine    col  labora tioiio    J./       Indeed,    from   a 

reading   of    the    published    literature    concerning    both 

maclii  ne-a  i  dcd    cognition   and    time-shoring,    one    gets    the 

impression  that  there  is  almost  unanimous  agreement  that  the 

use  of  such  systems  for  tills  'intimate  collaboration  between 

a  human  user  and  a  computer'  produces  far  more  pi'oductive, 

efficient,  and  effective  results  than  could  be  obtained  with 

any  other  type  of  system.   Those  persons  responsible  for 

major  developments  in  the  field  of  time-shared  systems 

believe  that  they  v/ill,  among  other  things; 

Improve  the  ability  to  program.  .  .open  up 
several  new  forms  of  computer  usage  6,/ 

increase  productivity  of  'computer  catalyzed 
research'  that  results  from  close  man-machine 
Interface.  .  .  2/ 

Unfortunately,  no  systematic  studies  arc  available  to 

support  the  above  statements.   There  appears  to  be  an 

emotional  commitment  to  these  systems  rather  than  a  formal 

investigation  of  existing  conditions.   Most  of  the  people  in 

the  field  have  apparently  relegated  formal  investigation  to 

a  low  priority  level.   in  fact,  the  first  published 
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ref>^ren'~e    to    this    problem    (and   one   which    did    not    appear 

ui..     :    .  miost    four    years    after   work    began    in    the    field) 

quBtloned   v;hether: 

The  promoter-scientists  of  time-sharing  are  sys- 
tematically, scientifically,  and  economically 
expl  or  i  ng  it ;  r,r   whet;iier  t  ime-shar  i  n;:^,  js  ROod 
,or  .bad  and  f'       what    purposes.   All  they  seem 
to  be  concer    v/ith  is,  which  kind  to  design, 
build,  or  bu;  pj    (underlining  added) 

In  addition,  at  tlic:  October  1965  meeting  of  the  Los  Angeles 

ACM  chapter  three  advocates  of  time-sharing  conceded  the 

absence  of  any  comparative  evaluation  of  time-shared  systems 

and  batch"process  I  ng  systems  9_/ . 

Some  of  those  persons  developing  time-shared  computer 
systems  have  attacked  proposals  for  formal  investigation 
Into  the  value  of  time-sharing  as  both  unwarranted  and 
incorrect.   They  advance  the  following  reasons  for  their 
attitude:  first,  time-shared  systems  are    so  newly  developed 
that  the  users  do  not  yet  know  how  to  mal;e  use  of  these 
systems;  second,  present  time-shared  systems  are 
experimental  in  nature  and  sliould  not  be  advanced  as 
representative  of  such  systems  of  the  future;  and  third,  it 
Is  alleged  tliat  time-sharing  applications  are  qualitatively 
different  than  batch-processing  applications  and  therefore 
cannot  be  compared  with  them. 

There  is  no  doubt  that  these  statements  are 
substantially  correct  •■-  time-shared  systems  are  new, 
developing,  and  different  than  the  more  traditional 
batch-processing  systems.   However,  It  is  not  necessarily 
true  that  the  stated  conclusions  follow  from  the  facts.   At 
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this  pointy  it  must  suffice  to  point  out  that  although  some 
tlme-'i^har  I  ng  applications  will  have  qualitative  differences 
when  compared  with  batch-processing  applications^  tlio 
majority  of  the  applications  are  expected  to  be  coinpotlble 
with  both  types  of  systems.   In  addition^  there  are  people 
v/ho  Jcnow  how  to  use  these  systems,  and  who  have  enough 
knovHedge  of  the  characteristics  of  future  eppl I  en t ions,  to 
design  an  Investigation  incorporating  at  least  some  of  tliem. 
These  aspects  v.'I  1  1  be  discussed  In  detail  belov.'. 
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.  of  Evaluating  Interactive  System  Usage  \^__ 

The  Interactive  time-shored  systems  arc  described  as 
'faster'/  'more  efficient'^  and  'more  productive'.  I.Q./  These 
termS/  as  used/  are  both  ambiguous  and  not  amenable  to 
measurement  unless  made  more  preci?c.   For  example/  'faster' 
could  refer  to  the  total  elapsed  t     froin  the  start  of  a 
problem  to  its  completion/  but  it     '.  d    also  refer  to  the 
total  amount  of  the  man's  time,  th    otal  amount  of  the 
computer's  time/  or  even  the  computer's  response  time.   The 
problem  Is  that  the  potential  evaluator  can  have  no  notion 
of  v.'hich  meaning  of  'faster'  is  relevant  to  the  persons 
using  the  term. 

As  another  example  of  this  problem/  a  prel    nary 
investigation  of  ojjji   measure  of  'efficiency'  f^     that  the 
use  cf  an  interactive  t  ime-sliar  i  ng  system  for  p.    cing  a 
v;orking  FORTRAN  program  resulted  in  a    six- fold  i.iciease  in 
computer  time  used  1.1/  (When  compared  with  the  lime  required 
for  the  same  problem  done  on  a  non-i  rite  met  i  ve  computer 
system).   However/  this  is  acceptable  re  'evidence  of  the 
Inefficiency'  of  interactive  t  ime-£.lit'M  i  ng  usage  only  if 
there  Is  agreement  between  the  relevant  cvaluatorn  tliot  the 
above  is  the  one  valid  measure  of  'efficiency'. 

Another  problem  that  appears  here  concerns  tlic 
determination  of  whom  the  relevant  evaluators  arc.   It  is 
doubtful  that  any  group  formed  solely  of  programmers/  users/ 
or  people  paying  for  the  computer  system  v/ould  be  acceptable 
as  evaluators  to  those  persons  to  whom  an  evaluation  might 
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be  addressed.   --  mixed  groups  would  doubtless  find  it 
difficult  to  agree  on  acceptable  measures. 

The  methodological  differences  between  computer 
applications  using  interactive  time-shared  systems  and  the 
more  traditional  computer  systems  result  in  further 
problems.   For  example,  the  programming  of  a  problem  on 
these  two  types  of  systems  can  be  expected  to  result  in  a 
different  allocation  of  time  for  problem  definition,  program 
writing,  statement  declaration,  debugging,  editing, 
compiling,  analysis,  waste  time^  think  time,  reacqua i ntance 
tima,  and  total  time  to  complete  the  problem. 

In  addition,  en  evaluation  could  be  furtlier  complicated 
by  attempting  to  compare  the  two  systems  by  using  a  problem 
v/fiich  is  not  fully  defined  on  the  traditional  system.   This 
could  result  from  an  application  v/hich  uses  features  of  an 
Interactive  time-shared  system  as  an  integral  feature  of  the 
tasi'v.   For  example,  a  tas!<  in  v/h  i  ch  the  user  and  the 
computer  v/ould  converse  frequently  v/i  th  each  other  fits  into 
this  category  because  it  depends  greatly  on  the  rapid 
response  available  with  a  time-shared  system.   Although  this 
tyiv:?  of  worl^  could  be  done  on  a  traditional  ba  tch- process  i  ng 
system,  the  large  amount  of  interaction  required  m.akes  it 
Imprec t i  cal . 

An  integral  factor  in  this  lack  of  formal  investigation 
Is  probably  prompted  by  the  (usually  found)  electrical 
engineering  and  computer  science  background  and  orientation 
of  those  persons  responsible  for  the  design  and  development 
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;f  i-.'-'C    Interactive  time-shared  systems.   Few,  If  any,  hove 
;,ac;  ;  behavioral  science  orientation  or  the  training 
necessary  for  deslcnlng  and  conducting  formal  Investigation 
to  evaluate  man-machine  interactive  aspects  of  the  usage  of 
these  systems. 

It  would  thus  appear  that  both  the  commitment  to  tlie 
time-sharing  concept  and  the  laclc  of  a  behavioral  science 
orientation  of  the  designers  and  developers  of  such  systems 
are  the  primary  factors  responsible  for  this  lack  of 
scientific  investigation  into  the  supposed  advantages  of 
Interactive  time-sharing. 
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Sc:iipjg._ /iM.  Pur P.Q-S^ 

The  above  suggests  four  possible  sets  of  variables 
relevant  to  an  investigation  of  this  area/  namely  those 
wlilch:  (1)  are  characteristic  of  the  computer  systems  under 
study;  (2)  concern  the  attitudes   of  the  users  of  these 
systems;   (3)  are  measures  of  the  user's  behavior  and  usage 
of  the  computer  system;  and  ('})  are  measures  of  the  results 
(output)  produced  by  the  man-computer  system. 

FIffure,  2, 
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Measures 


I  Evaluated  I 
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Figure  2  displays  the  conjectured  relationships  between 
these  four  sets  of  variables.   More  specifically, 
preliminary  experimentation  and  the  viewed  literature 
provides  support  for  the  following  major  conjecture: 


As  a  result  of  the  characteristics  of  a  time-shared 
computer  system,  the  user  of  such  a  system  will  have 
more  favorable  attitudes  toward  both  the  use  of  this 
computer  system  and  the  results  he  produces  wliile  using 
it  than   he  would  if  using  a  traditional  computer 
system.   Both  the  characteristics  and  the  ensuing  more 
favorable  user  attitude  will  result  In  the  user 
producing  results  faster  and  more  efficiently  than  he 
could  If  using  the  traditional  computer  system. 
Finally,  the  results  produced  by  the  user  exhibit  a 
quantitatively  higher  performance  level  than  the 
results  produced  if  a  traditional  computer  system  had 
been  used. 
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Before   discussing    each   of    the    above    sets   of    variables    in 
detail/     it    shoLild    be    noted    that    the    cor.jec tiTe    specifically 
refrains    from   stating    that   either    computer    system  v/i  1  1    be 
proven    'better'     than    the   other    "-    tliic    coi'lc!    bo    attempted 
only    if    'better'    v/as    interpreted    by    eac '       nd    every   user    in 
exactly    the    same  way.      However/    each    l!:  sslgns    different 

values    and    tre.de-offs    to    ii  i  s    requircmen         for    speed    of 
solution,    cost/    access    speed/    and    system    size    for    each    type 
of    application.      The    scope    of    this    study   will    be    limited    to 
Investigating    the    usage   of    both    time-shsred    end    the   tnore 
traditional    computer    systems    and    providing    a   means   of 
evaluating    the   use   of    either    type   of    sys tcru    (or    some 
combination   of    each)    over   a    range   Oi'    applications. 

£omPu.tiir_S^sj;.eni  £.h3.ra.c ie.ni  s.tlci3. 

The   major    independent    vari  abler-    to    be    conr !  dcred    in 
this    research   are    those   which    rcprosciit    the   dii  -iccr. 

between    Interactive    time-shared    comrti'icr    ryf-tcmc  the 

more    tradi  t ionej    computer    systems.       Th:;-"    v;;r!al  '  /i    be 

conveniently   categorized   as    those   wiilcli    rciiilt    f  (1) 

differing    feedback   delays;    and    (2)    c'iffcrinfi    c!  o^ 

Interaction    between    the    liser    and    tlic.    Cviivutcr    r;     'erne       In 
addition/     it    is    alleged    that    proper    itc   an   bo   made   of    these 
characteristics   of    interactive    timo-sIiCrcu    r-ystcmn    to 
produce   qualitatively   different    rcsponr-e    to    tlr..    tircr. 

The  elapsed  time  the  user  must  wait  from  submission  of 
his  program  to  the  return  of  results  is  a  measure  of  the 
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feedback  (^r-lns/_.   Interactive  tl  me-slicr  i  ng  systems  would  be 
expected  to  have  normal  feedback  delays  measurable  in 
seconds  while  the  traditional  systems  would  measure  this  in 
hours  or   days.   The  decree  of  1  v  ^•^'•pc,t..i.,on  between  the  user 
and  the  computer  can  be  measured  by  both  the  delay  involved 
before  a  request  for  service  is  recognized  and  the 
difficulties  a  user  encounters  in  making  one  extra  computer 
run.   Althouch  admittedly  i?  gross  measure^  we  shall  consider 
a  computer  system  as  Interactive  if  the  response  to  a  normal 
Input  is  expected  rapidly  enough  that  the  user  does  not  turn 
to  otiier  work  while  waiting  for  the  response  to  his  input. 
^12/  The  rapid  interaction  betv.'een  the  user  and  the  computer 
available  only  with  a  large  time-shared  computer  allows  the 
Implementation  of  a  programming  system  in  which  the  user  can 
take  an  active  part  In  the  computing-problem  solving  process 
--  tlie  man-machine  symbiosis  so  often  discussed, 

U^en  AtJiIiud.e.s. 

As  a  result  of  these  differences  in  the  characteristics 
of  the  computer  systems  under  i nvest igat i on^  it  is  expected 
that  a  user's  attitude  toward  botIi  the  computer  system  used 
oi:d  the  results  p:   ''iced  while  using  this  system  will 
differ.   In  parti     r,     it  is  expected  the  rapid  feedback 
feature  of  the  ti.     ,aring  system  will  increase  a  user's 
evaluation  of  the    fulness  of  both  the  computer  system 
used  and  the  results  of  any  given  interaction  with  the 
computer  system.   In  addition,  the  ability  to  Immediately 
correct  errors  and  re-run  a  program  available  with 
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:ln       i ng  should  result  in  a  smaller  change  in  a  user's 
level  of  satisfaction  after  an  unsuccessful  computer  run 
than  if  the  user  had  used  a  traditional  computer  system, 
f'or  the  same  reason.  It  is  expected  that  a  time-sharing 
user's  estimate  of  progress  toward  completion  of  the  project 
would  be  less  affected  by  an  unsuccessful  run  than  would  the 
estimate  of  a  user  of  a  traditional  computer  system. 

(1)  The  user  of  a  time-sharing  system  v'll  rate 
the  usefulness  of  both  the  computer  sy:     and 
the  results  of  a  computer  run  higher  t'    i/i  1  1 
the  user  of  a  traditional  system. 

(2)  The  ne/7;ative  influence  of  an  unsuccessful 
computer  run  on  a  user  v;i  1  1  be  stronger  in  a 
batch-processing  than  a  time-sharing 

envi  ronment . 

lJ.Siir_Bj£hiivloj:  a.nd.  iisa.g£.   P.ejlfG.i'OiaiLce  lle^su.r£.s 

When  evaluating  usap:e  of  a  man-computer  system^,  we  are 
concerned  with  both  the  amount  of  elapsed  time  required  to 
obtain  a  solution  to  a  problem  and  tlie  efficiency  with  which 
this  solution  could  be  ob  tal  ned .  j_3/  Experience  with  both 
time-sharing  and  the  more  traditional  computer  usage  leads 
us  to  expect  tlie  faster  fcedbacJi  available  with  a 
time-sharing  system  will  result  in  a  problem  solution 
requiring  a  shorter  elapsed  time  for  completion. 

The  man- computer  systems  under  study  will  also  be 
evaluated  by  measuring  both  components  of  the  cost 
efficiency  and  the  methods  of  usage  of  the  two  systems.   The 
cost  efficiency  Is  determined  by  calculating  the  cost  of 
both  the  man's  time  and  the  computer's  processing  time  for 
each  system.    We  would  expect  that  usage  of  a  time-sharing 
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system  will  re;  it  in  a  lower  cost  of  the  man's  time  and  a 
higher  cost  o:      computer's  time.   The  aqs t  .ef f i  ci  encv 
will  depend  up.   the  balance  between  the  savings  resulting 
from  a  reduced  number  of  man-hours  and  both  the  increased 
number  of  hours  of  computer  processing  time  consumed  and  the 
higher  cost  of  time-sharing  usage. 

The  method  of  usage  is  a  measure  of  the  degree  to  which 
the  user  relegated  programmable  problem  solving  to  the 
computer  system»   As  a  result  of  the  user's  attitude 
concerning  the  access  i  I.)  i  1  i  ty  and  usesbility  of  time-sharing, 
we  would  expect  the  user  to  relegate  a  larger  number  and 
percentage  of  the  programmable  tasks,  such  as  checking  for 
errors,  to  the  computer  while  concentrating  his  attention  on 
non-programmable  tasks,  such  as  analysis  of  logical  errors 
and  deci  slon-malcing.   This  does  not  really  mean  that  such 
usage  Is  more  or  less  efficient  than  usage  of  a  different 
computer  system.   Rather  it  indicates  differential  modes  of 
behavior  while  using  these  systems  --  behavior  v.'liich  may 
influence  the  final  results  produced. 


ilQJiig/:     ■  (1)   The  use  of  a  time-sharing  computer  system 
will  result  in  a  task  solution  sooner  than  the 
solution  could  be  obtained  with  a 
batch-processing  computer  system. 

(2)   The  use  of  a  time-sharing  computer  system 
will  result  in  a  greater  amount  of  computer 
processing  (and  higher  computer  processing 
cost)  tiian  corresponding  usage  of  a 
batch-processing  computer  system.   However, 
this  usage  of  a  time-sharing  computer  system 
v;ill  require  fewer  man-hours  than  for  a 
corresponding  ba  tcli-process  i  ng  computer  system. 
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(3)   The  differential  user  attitudes  toward  the 
two  types  of  computer  systems  will  result  in 
greater  use  of  the  time-shared  system  -- 
however,  this  greater  use  will  result,  in  part, 
from  a  greater  reliance,  by  the  user,  on  the 
computer  for  routine  tasks. 

£xl.ejin^l_E^alu^tioa  af_Pje.t*£oj:ma.n£.e 

The  purpose  of  the  external  evaluation  of  performance 
Is  to  make  available  a  measure  of  the  output  of  the 
man- computer  system  which  is  arrived  at  independently  of  the 
user's  methods  or  the  computer  system  used.   To  accomplish 
this  objective  both  the  results  produced  by  the  user  on  the 
computer  system  and  an  evaluation  (by  a  competent  evaluator) 
of  the  user's  understanding  of  the  problem  and 
Implementation  of  a  solution  will  bo  used  to  judge 
performance. 

The  published  literature  suggests  that  time-sharing 
usage  will:  "improve  the  ability  to  program";  "open  up 
several  now  forms  of  computer  usage";  and  "increase  (the) 
productivity  of  'cotnputer  catalyzed  research,..".   From 
preliminary  experimentation  it  appears  that.  In  fact,  the 
use  of  time-sharing  .does  allow  the  user  to  produce  'better 
results'  ""  If  the  task  In  question  is  to  analyze  a  problem 
and  produce  a  decision-rule,  the  user  of  the  time-sharing 
system  will  produce  a  better  decision  rulr.   If  the  task  Is 
to  produce  a  workxing  computer  program,  the  user  will  produce 
a  program  In  which  results  are  more  often  correct,  have  done 
more  checking  of  inconsistencies  in  the  data,  or  is  more 
completely  documented  than  a  program  produced  by  a  user  of  a 
batch-processing  computer  system. 
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The  results  produced  by  the  user  of  the 
tinie-sliorl  ng  system  will  exhibit  a  higher  level 
of  perfcrmance  than  the  user  of  a  traditional 
computer  system. 
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Experimental  DosUn 
l'U.r(aducI.ian 

The  selection  of  an  appropriate  research  setting, 
experimental  tasi;,  and  experimental  subjects  is  important  in 
every  investigation  of  tli  i  s  type.   Since  one  purpose  of  thhs 
research  is  to  provide  answers  to  questions  concerninj;^  the 
value  of  time-sharing  usage,  the  research  settings  and  taslcs 
selected  should  have  relevance  for  both  '  rea  1 -v;or  1  d  ' 
programmers  and  managers  in  terms  of  their  potential  uses  of 
time-sharing  systems.   In  addition,  the  experimentation  must 
be  conducted  with  subjects  v/ho  are  competent  users  of  these 
types  of  computer  systems. 

Three  research  settings  at  i-i.l.Tc  were  selected  as 
appropriate  in  terms  of  the  above  ci'iteria:  (1)  a  graduate 
course  in  industrial  Dynamics,  given  by  the  Sloan  School  of 
Management;  (2)  an  advanced  undergraduate  programming 
course,  given  by  the  Electrical  Engineering  Department;  and 
(3)  an  introduction  course  In  management  information  systems 
also  given  by  the  Sloan  School  of  flanagement .   The  first  and 
second  courses  require  the  students  to  program,  debug,  and 
analyze  problems  using  the  MIT  Computation  Center's  IBM  709^1 
computer  which  is  used  for  both  time-sharing  and 
batch-processing  applications.   The  ttilrd  course  makes  use 
of  the  Sloan  School's  non-time  shared  IBM  1620  computer  for 
the  same  typo  of  work. 

The  Industrial  Dynamics  course  was  Included  in  this 
study  because  industrial  Dynamics  requires  the  use  of  a 
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or...  ■.e-.y    system  for  the  primary  purpose  of  analyzing  the 
..haracteri  s  t  i  cs  of  computer  simulation  models  and  dosli^ning 
(Vianagement  policies  based  on  their  analysis.   The 
undergraduate  programming  course  was  included  to  investigate 
the  behavior  of  experienced  programmers  working  on  a  fairly 
well  defined  problem  which  primarily  required  programming 
effort.   The  management  information  systems  course  v;as 
included  because  it  would  allow  an  investigation  of  the 
effect  of  providing  a  'better'  quality  and  different  amounts 
of  computer  response  to  the  user  while  maintaining  the  same 
feedback  delay  -•-  a  feature  not  found  in  the  other  two 
setti  ngs. 

£x2.eii  i  mejD.  t  a  1  _T  a.sJis 

The  tasic  to  be  used  In  the  I  nc'     ial  Dynamics  course 
assignment  will  involve  the  analysi;   'f  market  dynamics  in  a 
'Construction  Industry'.   Eacli  student  in  the  course  will  be 
required  to  talce  the  role  of  a  small-scale  Independent 
builder  within  this  industry  who  is  trying  to  maximize  his 
average  profits  in  light  of  certain  market  dynamics. 
Through  his  analysis  of  the  market  and  its  behavior^  the 
student  is  expected  to  arrive  at  a  decision  rule  for 
starting  apartment  construction  that  will  accomplish  this 
goal . 

The  task  for  the  industrial   Dynamics  course  will  be 
administered  through  the  use  of  a  'case'  assigned  as  a 
homework  problem.   The  actual  problem  will  be  botli  simple 
and  higiily  constrained,  v/i  th  the  construction  of  the 
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Jnorpendent  builder's  decision  rule  to  maximize  average 
prcvlts  the  primary  area  for  creative  v^orlu   Tlie  students 
will  be  required  to  complete  the  programminc  of  a  simulation 
model  of  this  'case'  (using  the  DYNAMO  programming  language 
IJi/)/  debug  the  program,  analyse  the  results,  end  propose 
and  test  a  decision  rule  to  be  used  by  the  independent 
bul Ider. 

The  assignment  for  the  advanced  programming  course  will 
be  the  programming  of  a  theorem  proving  algorithm  in  the 
field  of  mathematical  logic  using  the  MAD-SLIP  programming 
system  available  on  the  IBM  7091^. 

The  management  information  systems  course  will  require 
the  programming  of  an  accounting  and  inventory  control 
assignment  for  a  manufacturing  company^   The  student  will  be 
assigned  to  one  of  the  two  FORTRAN-compat i b 1 e  programming 
languages  available  on  the  non-time  shared  I  Bf'1  1620  computer 
and  will  be  allowed  to  program  the  problem  in  any  way 
des I  red. 

The  students'  primary  concern  In  the  three  assignments 
will  be  with  the  course  project  required  by  the  Instructor 
-"  the  data  collection  questionnaires  will  be  explained  as 
an  outside  research  pi"oject  having  no  Influence  upon  their 
grade  for  the  assignment.   As  a  result,  tlie  students  should 
be  primarily  motivatc-d  by  the  class  assignment  "-  a  stronger 
motivation  to  participate  than  that  provided  by  most 
laboratory  experiments  which  use  a  small  sum  of  money  as  the 
pr  imary  mot  I vator. 
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MethouolQ>;y 

The  general  methodology  for    all  three  experimental 
tasks  will  be  described  in  two  distinct  phases:  (J.)  the 
procedure  for  conducting  the  research;  and  (2)  the  data  to 
be  collected  aiid  the  method  of  collecting  It.   in  addition, 
a  description  of  the  Industrial  Dynamics  pilot  study  v;i  1  1  be 
discussed. 

£xj2e£.l  men  t^l  _Pi:o^eciure 

The  experimental  tasks  will  be  presented  to  the 
students  as  part  of  the  normally  required  course  assignment 
for  all  three  courses.   While  assigning  the  problem^  the 
instructor  v.'i  1  1  notify  the  class  that  a  research  group  (in 
tlie  case  of  the  Industrial  Dynamics  course,  the  Industrial 
Dynamics  Researcli  Group)  is  involved  in  a  study  of  some 
aspects  of  computer  usage  and  will  be  collecting  data 
concerning  the  methods  used  by  the  students  in  completing 
assignments  of  the  type  being  assigned  to  the  class.   Before 
the  assignment  is  presented  to  the  students,  they  will  be 
aslced  to  fill  out  pre~exper  imental  questionnaires  (see  the 
next  section  for  a  discussion  of  the  data  to  be  collected 
through  the  use  of  this  questionnaire).   Based  on  their 
answers^  students  with  similar  backgrounds  will  be  paired 
and  each  assigned  to  a  different  experimental  treatment  -•■ 
this  assignment  will  be  done  on  a  random  basis. 

Each  student's  assignment  to  either  the  time-sharing  or 
batch-processing  computer  systems  (or,  in  the  case  of  the 
management  information  systems  course,  either  of  the  two 
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FORTRAN  compatible  programining  systems)  v/il  1  be  handed  out 
along  with  the  course  project  material.   During  the 
assignment,  and  before  the  students  begin  using  the 
computer/  they  will  have  several  questionnaires  to  complete. 
In  addition,  after  each  use  of  the  computer,  the  students 
will  be  required  to  complete  a  short  questionnaires 
pertaining  to  that  interaction  with  the  computer. 

The  students  using  the  batch-processing  system  will  be 
able  to  submit  their  computer  runs  once  or  twice  each  day  -- 
depending  on  the  course  they  are    in.   The  students  assigned 
to  the  time-shared  system  will  be  able  to  use  the  computer 
at  almost  any  hour  of  the  day  or  evening.   For  the 
Industrial  Dynamics  course,  the  students  will  have  four 
consoles  available  for  their  exclusive  use  --  all  four 
consoles  will  be  continuously  'logged  in'  to  the  time-shared 
system  and  the  students  w-I  1  1  have  access  limited  only  by  the 
other  students  using  the  consoles  for  the  same  purpose..!.^./ 
For  the  advanced  programming  course,  the  students  will  be 
able  to  'log  in'  to  the  computer  from  a  console  anywhere  on 
t:.  ;  M.I.T.  campus.   In  the  case  of  the  management 
information  systems  course,  time  on  the  Sloan  School's  IBM 
1G20  computer  will  have  been  reserved  for  several  periods 
during  the  day  in  addition  to  normally  schedul cd  .servl ce 
runs. 

D.a_t.a_C(ilie£.tioa 

The  data  will  be  collected  by  both  the  questionnaires 
completed  by  the  students  during  the  assignment  and/or  of 
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th'"  conij-'Uter  Input  and  output  of  each  student.   This  will  be 
dor.i.  to:  (1)  discover  how  each  student  works;  ancJ  (2)  cross 
check  the  data  obtained  from  the  ques ionna i res  the  students 
compl eted. 

These  questionnaires  will  take  several  forms:   First  a 
pre-exper Imen tal  questionnaire  to  gather  background 
information  before  any  manipulations  take  place.   Second,  a 
ques t iorma i re  administered  before  the  students  make  use  of 
the  computer  will  investigate  tfie  influence  of  knowing  that 
they  will  be  using  one  or  another  computer  system.   Third,  a 
questionnaire  completed  after  their  analysis  of  each 
computer  run  to  investigate  the  students'  attitudcis  and 
behavior  during  the  course  of  the  experitnent  and  to  obtain 
measures  of  progress/  time  expended,  and  the  type  of  work 
done.   The  fourth  type  of  questionnaire,  administered  after 
the  students  have  completed  the  assignment,  will  investigate 
the  above  factors  on  a  overall  basis  as  v/cll  as  requesting 
comments  concerning  the  assignment.   The  final 
questionnaires  will  be  administered  three  weeks  after  the 
experiment  and  will  measure  the  degree  of  interaction 
betv/een  the  students  while  solving  this  problem. 

The  data  to  be  collected  through  questionnaire  and/or 
content  analysis  of  the  printed  output  will  be:  (1)  the 
amount  of  time  expended  on  the  experimental  task  and  the 
distribution  of  that  time  over  the  period  the  experiment  is 
in  progress;   (2)  the  categorization  of  this  expend    'e 
into  analyzing  results,  making  changes,  v;aiting  for      rn 
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ofrc-Lults^  etc.;   (3)  the  number  of  computer  runs 
submitted;  ih)    analysis  of  types  of  changes  made  and  the 
reasons  for  these  changes;  (5)  analysis  of  types  of  errors; 
(6)  student's  evaluation  of  the  'usefulness'  of  each  run's 
results;  and  (7)  student's  evaluation  of  'effectiveness'  of 
each  run'  s  resul  tr>. 

£llol  Stiidji 

One  pilot  study  has  been  conducted  to  investigate  the 
use  of  the  particular  Industrial  Dynamics  task  (  th(?  'case' 
study  of  the  Construction  Industry)/  and  the  use  of  the  data 
collection  devices.   For  the  pilot  experiment,  several 
students  who  had  previously  taken  the  Industral  Dynamics 
cours'    /  were  asked  to  participate  in  a  pre-test  of  the 
experi    t.   After  completing  the  experimental  task  they 
were  asked  for  individual  comments  concerning  the 
presentation  of  the  experiment,  the  interest  generated  by 
the  task/  and  the  data  collection  devices  used.   All 
pro-test  subjects  were  requested  to  refrain  from  discussing 
the  experiment  with  anyone  —  it  appears  as  if  this  request 
v/as  honored. 
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Spm '■■  P o '-, ?  I  b  1  e  Ex i^ p r  i  n'lontfll  Difficulties. 

The  following  are  several  potent.ii'l  problems  inherent 
in  the  design  of  the  proposed  researcli:   Tir^t,  tlio 
appropriateness  of  botli  the  experimental  settings  and  the 
experimental  tasks  could  be  questioned  ~-  especially  by 
those  v/Iio  believe  that  time-sharing  applications  have  not 
yet  been  developed  and,  when  tfiey  have  been  developed,  will 
be  qualitatively  different  than  traditional  applications. 
This  assumes  that  the  area  of  time-sharing  and 
batch-processing  applications  do  not  and  will  not  overlap. 
In  fact,  experience  to  data  on  several  operating  systems  has 
shown  many  areas  where  this  overlap  is  substantial.  1 7/ 

The  possibility  of  conrnuni  ca  t  i  on  from  one  subject  to  a 
future  subject  Is  a  potential  problem  in  the  conduct  of  the 
experimental  tasks,   Wliether  the  content  of  the 
communication  had  value  o,r  not,  the  conduct  or  actions  of 
the  other  subjects  in  the  experimental  situation  might  be 
affected  in  some  way.   Although  emphasizing  the  importance 
of  independent  work  to  the  students  may  not  insure  they  work 
Ir     ndently  of  one  another,  measures  taken  after  the 
s:     ts  receive  their  grades  v/i  1  1  allow  for  control  of  this 
p :    um , 

The  last  Important  area  of  potential  problems  concerns 
data  collection  and  the  filling  out  of  the  'Data  Sheet'  by 
the  subjects  participating  in  the  first  tas'k  after  each 
computer  run.   There  may  bo  some  problems  connected  with 
requiring  the  subjects  to  remember  the  necessary  dat-'. 
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However/  the  only  alternative  to  the  proposed  method  of  data 
collection  is  the  use  of  a  time  record,  divided  into  ten  or 
fifteen  minute  intervals,  which  the  subjects  would  fill  out 
during  each  interval.   The  'Data  Sheet'  method  was  chosen  to 
eliminate  the  possibility  of  subject  antagonism  resulting 
from  the  large  amount  of  time  required  and  both  Intentional 
and  unintentional  errors  and  biases  introduced  as  a  result 
of  using  the  second  method. 
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Summarv  and  Implications 

To  briefly  SL'mmai' ize  the  major  hypotheses  of  this 
research,  it  has  been  predicted  that  the  user  of  a 
time-sharing  computer  system  will  demonstrate  a  higher  level 
of  performance  than  will  the  user  of  a  more  traditional 
computer  system.   Furtlier/  it  was  predicted  that  this  higher 
level  of  performance  will  result  from  differential  usage  of 
the  two  types  of  computer  systems  and  that  the 
characteristics  of  each  computer  system  affects  the  user's 
attitudes  tov.'ard  the  computer  system. 

The  major  potential  implication  of  this  study  is  that 
pe'rsons  responsible  for  the  design  and  selection  of  computer 
and  information  systems  should  consider  tiie  trade-offs  among 
their  requirements  for  speed  of  solution,  cost,  access 
speed,  and  size  of  the  accessible  system  before  determining 
the  type  of  computer  system  to  be  used. 

In  addition,  it  is  hoped  ti<at  this  study  v/i  1  1  provide  a 
basis  for  evaluating  time-siiar  i  ng  and/or  interactive 
computer  systems  whicli  have  been  designed  to:  (1)  !-■    'de  an 
ni'-.sdiate  'debugging'  tool  for  tlie  programmer;  (2)  ,     de 
the  researcher  with  the  ability  to  test  conjectures 
hunches  at  the  time  they  occur;  (>)    provide  the  tea-'ier  with 
a  tool  for  immediate  response  to  student  inquiries;  and  ('t) 
provide  a  system  to  tiie  manager  for  the  purpose  of 
experimenting  with  several  different  policies  in  a  simulated 
environment  so  the  policy  exhibiting  the  desired 
characteristics  can  be  implemented. 
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It  is  Important  to  realize  that  •  organization's 

requirements  for  a  computer  or  infon  "on  rystem  may  be 

such  that  implementation  of  a  tima"s!  g  system  would  not 

only  increase  the  total  cost  of  a  cc  '  system,  but  also 

downgrade  the  service  previously  pro  with  a  traditional 
computer  system. 
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FOOTNOTES 


1  Tv;o   definitions    of    time-sharing    by    experts    in    the    field 
are:    "i^y    a    time-sharing    computer    sy5>tem    1    shall    mean   one 
that    interacts   with    many    simultaneous    users    tlirough    a 
number    of    remote    consoles.       Such    a    system  will     look    to 
eacli    user    like    a    large    private   computer".      McCar  thy  (  32) . 
Also/    "I3y    time-sharing   we   moan    those    systems    in   which    the 
facilities    of    a    computer    comijlex    are    rapidly    commutated 
among    Independent    users   who   are    eacli    on-line    at    a    remote 
console."      Glaser    and    CorbatcS    (20).       The    terms 
'Interactive'    and    'time-shared'    will    be    used 
Interchangeably    in    the    following    discussion.       It    should 
be    realized,     that    although    time-shared    computer    sys'     •;:, 
are    Interactive,     the    reverse    Is    not    necessarily    true.       We 
are    cons  I  der  I  nr:..     in    this    discussion,    comi:iULcr    systems 
which    are    botli    interactive    and    time-shared. 

2  See    McCarthy    (32),     Corbati    (G),     Fano    (  U. ) ,     and    Licklidor 
(27)    for    a    discussion    of    the    chatiges    that    have    resulted 
from    th i s    deve lopment . 

3  ■  The  Arnorican  £ederatlon    of   Information  £roco5sing 

jSocI  Cities    Is    composed    of    three    primary    groups;     tlie 
Association    for    Computing   Machinery,     the    liZlir,    aiid    the 
American    Ooc""''Tita  t  i  on    Institute. 

l»      Tlie    ad>Jitional    "out    of    pocl;ct"    expense    required    for 

time-sharing    results    from    the    extra    computer    h.jrdwdre    and 
software    required,     the    time-sharing    accounting,     tiie 
transmission    lines    and    user    consoles,    and    the    system 
tna  I  ntal  nance.       Iloweve-r,    since    the   metliod    of    usoi^e   of 
tliese    systems    is    presently    unknown,    any    costs    or    savin./., 
resulting    from    this    usage    cannot    be    calculated. 

5      Gee    LIcklider    (27),    Corbati    (G),    Schwartz    CiO),    and    Slot/. 
( i»  :i ) . 

C  Corbat6,    Daggett,    and    Daley (7). 

7  Dennis    (11). 

8  f-ein    (13),     p.     I's. 

9  (!j),    p.     10  3. 

10  IVhen    asked    to    compare    interactive    time-sliarod    systems 
with    ba  tcli-process  I  ng    computer    systems,     both    part-time 
and    professional     programmers    were    almost    unanliiious    in 
tlieir    agreement    that     Interactive     time-shared    -..ystems    are 
'better'.       When    pressed    to    define    'better',     tlicy    say 
first    that    they    are    able    to    do    a    'better'    job    on    sucli 


PAGE  29 


-systems,  thus  increasing  their  satisfaction.   Tlicy  also 
point  out  that  the  time  to  complete  this  job  is  less  than 
It  would  have  been  had  they  used  a  batch-processing 
system.   These  programmers  do  not  explicitly  cite  other 
measures  of  productivity/  efficiency,  or  effectiveness. 
(These  observations  resulted  from  informal  contacts  v>'I  th 
professional  time-sharing  programmers  at  M.I.T.  and 
Systems  Development  Corporation.) 

11  K'hen  the  end  products  (the  working  programs)  were 
evaluated,  there  v^as  no  apparent  difference  in  the  final 
programs  produced.   There  was,  however,  a  difference  in 
the  total  elapsed  time  to  produce  these  progranis.   There 
seem  to  be  two  components  causing  this  Increase;  first, 
the  use  of  the  Project  MAC  time-sharing  system  required 
twice  as  much  processor  time  for  printing  and  editing 
programs  as  for  compiling  them  --  a  three-fold  increase 
In  processor  usage;  and  second,  the  behavior  of  the  users 
apparently  changed  as  a  result  of  the  greater 
availability  of  the  computer  with  time-shared  systcins. 
This  resulted  In  a  greater  number  of  compilations  (and 
the  accompanying  Increase  in  printing  and  editing)  than 
vsfould  have  occurred  with  the  traditional  batch-processing 

"system  --  this  increase  have  been  variously  estimated 
with  estimates  ranging  bet\'  n    50?o  and  i^OO!^. 

12  It  should  be  noted  that  th i  'eflnltion  Is  relevant  only 
in  terms  of  on-line  system  traditional 
batch-processing  systems  a.  y  their  very  nature  and 
design,  non-interactive. 

13  We  arc   mailing  the  simplifyi,   assumption  that  the 
solution  could,  in  fact,  i^e  ubtalned.   We  could.  If 
necessary,  treat  this  as  another  variable. 

l'>  The  DYNAMO  programming  language  is  described  in  Pugh  (3  7) 
and  Forrester  (17).   Industrial  Dynamics  is  described  in 
Forres  tor  ( 17'; . 

15  Preliminary  experimentation  indicates  this  number  of 
available  consoles  to  be  sufficient  to  handle  tlie 
students'  requirements  of  access. 

16  We  are  assuming  that  these  students  have  the  same 
qualifications  as  those  students  who  will  participate  in 
the  actual  experiment  —  with  the  exception  of  having 
completed  a  ful  course  in  Industrial  Dynamics. 

17  We  are  referring  to  the  experiences  of  two  experimental 
time-sharing  systems..  Project  MAC  at  M.I.T.  and  TSS  at 
System  Development  Corporation,  and  two  commercial 
time-sharing  systems,  the  IBM  QUIKTRAN  system  and  the 
KeyData  system. 
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